Enable RVV-optimized TRSM kernels for RISCV64_ZVL128B#5830
Draft
Felix-Gong wants to merge 1 commit into
Draft
Conversation
Use existing RVV-optimized TRSM kernel implementations instead of the
generic C versions for the RISCV64_ZVL128B target. The RVV kernels
(trsm_kernel_{LN,LT,RN,RT}_rvv_v1.c) are already present in the
repository and used by the x280 target, but were not enabled for
ZVL128B.
Note: ZVL256B is not included because the RVV TRSM kernel has
correctness issues with VLEN=256.
Signed-off-by: Felix-Gong <gongxiaofei24@iscas.ac.cn>
Felix-Gong
force-pushed
the
feature/rvv-trsm-zvl128b
branch
from
June 7, 2026 03:43
c50f0a6 to
4fb51e3
Compare
Felix-Gong
changed the title
Felix-Gong
marked this pull request as draft
June 7, 2026 08:13
Contributor
Root-cause analysis of the
|
| type | SEW | VSETVL_MAX (solver M-block) |
GEMM_UNROLL_M (packer) |
match? |
|---|---|---|---|---|
| S | 32 | 16 | 8 | ✗ |
| D | 64 | 8 | 8 | ✓ |
| C | 32 | 16 | 8 | ✗ (→ cblas_ctrsm FAILED) |
| Z | 64 | 8 | 4 | ✗ |
This lines up exactly with the observed cblas_ctrsm failure, and it is the same effect as the "wrong results at VLEN=256" that motivated excluding ZVL256B — it's one root cause, not two. x280 is unaffected because its GEMM kernel and copies are all _rvv_v1, so the rectangular block width is also VSETVL_MAX and everything agrees.
Note the mismatch depends on the runtime VLEN, not the build target name: a ZVL128B binary run on a VLEN≥256 core still uses VSETVL_MAX≥16, so it breaks even though the target says 128.
Options
- Revert TRSM to generic for this target (safe, guaranteed green, drops the RVV TRSM speedup):
[SDCZ]TRSMKERNEL_{LN,LT,RN,RT} = ../generic/trsm_kernel_{LN,LT,RN,RT}.c - Keep the optimization by making the whole
ZVL128BGEMM path adaptive_rvv_v1(likex280), or by writing TRSM kernels that block byGEMM_UNROLL_Minstead ofVSETVL_MAX. This is a larger change and needs on-hardware verification that GEMM/TRMM don't regress.
I'm planning to verify both empirically on a SpaceMiT X60 (VLEN=256) — reproduce the ctrsm failure as-is, then confirm the chosen fix. Happy to push the fix once validated. Let me know which direction you'd prefer.
Contributor
|
Hardware test results on X60 (VLEN=256, GCC 14.3.0) I ran this PR through the ctest suite on real RISC-V hardware. The current PR fails — Root cause: the I verified this isn't just missing copy files: adding the Suggested path for this PR: revert the 16 (CTEST_RC=0, zero failures across the suite.) Optimized RVV TRSM on ZVL128B is worthwhile, but it is a complete level-3 ecosystem migration (self-consistent rvv_v1 GEMM + copies + params, matching the x280 target) rather than a 16-line kernel swap — better as dedicated follow-up work with its own hardware validation than folded into this PR. |
Contributor
On-hardware validation on X60 (ZVL128B): current kernels are VLEN-dependent and corrupt on non-x280 targets — fix + evidenceTested this PR on real RISC-V hardware (SpaceMiT X60, Root causeThe four Evidence (original vs fixed, same harness)Residual test DTRSM on ZVL128B:
Failures cluster exactly where Full fixed-kernel validation, all green on ZVL128B (8000 cases, 0 fails):
FixKeep the RVV-vectorized A patch against all four Validated on SpaceMiT X60 hardware. Patch: VLEN-agnostic outer control for trsm_kernel_{LN,LT,RN,RT}_rvv_v1.cdiff --git a/kernel/riscv64/trsm_kernel_LN_rvv_v1.c b/kernel/riscv64/trsm_kernel_LN_rvv_v1.c
index 869561fb3..fcdd4f3b9 100644
--- a/kernel/riscv64/trsm_kernel_LN_rvv_v1.c
+++ b/kernel/riscv64/trsm_kernel_LN_rvv_v1.c
@@ -71,6 +71,30 @@ static FLOAT dm1 = -1.;
#define GEMM_KERNEL GEMM_KERNEL_N
#endif
+#if GEMM_DEFAULT_UNROLL_M == 1
+#define GEMM_UNROLL_M_SHIFT 0
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 2
+#define GEMM_UNROLL_M_SHIFT 1
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 4
+#define GEMM_UNROLL_M_SHIFT 2
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 6
+#define GEMM_UNROLL_M_SHIFT 2
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 8
+#define GEMM_UNROLL_M_SHIFT 3
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 16
+#define GEMM_UNROLL_M_SHIFT 4
+#endif
+
#if GEMM_DEFAULT_UNROLL_N == 1
#define GEMM_UNROLL_N_SHIFT 0
#endif
@@ -91,7 +115,9 @@ static FLOAT dm1 = -1.;
#define GEMM_UNROLL_N_SHIFT 4
#endif
-// Optimizes the implementation in ../arm64/trsm_kernel_LN_sve.c
+// Packed A is tiled by GEMM_UNROLL_M (the GEMM micro-kernel / itcopy contract),
+// NOT by runtime VL: tiling by VSETVL_MAX only matches x280 and silently
+// corrupts on other targets (e.g. zvl128b, unroll 8). Only solve() vectorizes.
#ifndef COMPLEX
@@ -217,10 +243,6 @@ int CNAME(BLASLONG m, BLASLONG n, BLASLONG k, FLOAT dummy1,
BLASLONG i, j;
FLOAT *aa, *cc;
BLASLONG kk;
-
- size_t vl = VSETVL_MAX;
-
- //fprintf(stderr, "%s , %s, m = %4ld n = %4ld k = %4ld offset = %4ld\n", __FILE__, __FUNCTION__, m, n, k, offset); // Debug
j = (n >> GEMM_UNROLL_N_SHIFT);
@@ -228,63 +250,62 @@ int CNAME(BLASLONG m, BLASLONG n, BLASLONG k, FLOAT dummy1,
kk = m + offset;
- i = m % vl;
- if (i) {
- aa = a + (m - i) * k * COMPSIZE;
- cc = c + (m - i) * COMPSIZE;
+ if (m & (GEMM_UNROLL_M - 1)) {
+ for (i = 1; i < GEMM_UNROLL_M; i *= 2){
+ if (m & i) {
+ aa = a + ((m & ~(i - 1)) - i) * k * COMPSIZE;
+ cc = c + ((m & ~(i - 1)) - i) * COMPSIZE;
- if (k - kk > 0) {
- GEMM_KERNEL(i, GEMM_UNROLL_N, k - kk, dm1,
+ if (k - kk > 0) {
+ GEMM_KERNEL(i, GEMM_UNROLL_N, k - kk, dm1,
#ifdef COMPLEX
- ZERO,
+ ZERO,
#endif
- aa + i * kk * COMPSIZE,
- b + GEMM_UNROLL_N * kk * COMPSIZE,
- cc,
- ldc);
- }
-
- solve(i, GEMM_UNROLL_N,
- aa + (kk - i) * i * COMPSIZE,
- b + (kk - i) * GEMM_UNROLL_N * COMPSIZE,
- cc, ldc);
+ aa + i * kk * COMPSIZE,
+ b + GEMM_UNROLL_N * kk * COMPSIZE,
+ cc,
+ ldc);
+ }
- kk -= i;
+ solve(i, GEMM_UNROLL_N,
+ aa + (kk - i) * i * COMPSIZE,
+ b + (kk - i) * GEMM_UNROLL_N * COMPSIZE,
+ cc, ldc);
+ kk -= i;
+ }
+ }
}
- int mod = i;
- i = vl;
- if (i <= m) {
- aa = a + (m - mod - vl) * k * COMPSIZE;
- cc = c + (m - mod - vl) * COMPSIZE;
+ i = (m >> GEMM_UNROLL_M_SHIFT);
+ if (i > 0) {
+ aa = a + ((m & ~(GEMM_UNROLL_M - 1)) - GEMM_UNROLL_M) * k * COMPSIZE;
+ cc = c + ((m & ~(GEMM_UNROLL_M - 1)) - GEMM_UNROLL_M) * COMPSIZE;
do {
if (k - kk > 0) {
- GEMM_KERNEL(vl, GEMM_UNROLL_N, k - kk, dm1,
+ GEMM_KERNEL(GEMM_UNROLL_M, GEMM_UNROLL_N, k - kk, dm1,
#ifdef COMPLEX
ZERO,
#endif
- aa + vl * kk * COMPSIZE,
+ aa + GEMM_UNROLL_M * kk * COMPSIZE,
b + GEMM_UNROLL_N * kk * COMPSIZE,
cc,
ldc);
}
- solve(vl, GEMM_UNROLL_N,
- aa + (kk - vl) * vl * COMPSIZE,
- b + (kk - vl) * GEMM_UNROLL_N * COMPSIZE,
+ solve(GEMM_UNROLL_M, GEMM_UNROLL_N,
+ aa + (kk - GEMM_UNROLL_M) * GEMM_UNROLL_M * COMPSIZE,
+ b + (kk - GEMM_UNROLL_M) * GEMM_UNROLL_N * COMPSIZE,
cc, ldc);
- aa -= vl * k * COMPSIZE;
- cc -= vl * COMPSIZE;
- kk -= vl;
-
- i += vl;
- } while (i <= m);
+ aa -= GEMM_UNROLL_M * k * COMPSIZE;
+ cc -= GEMM_UNROLL_M * COMPSIZE;
+ kk -= GEMM_UNROLL_M;
+ i --;
+ } while (i > 0);
}
-
b += GEMM_UNROLL_N * k * COMPSIZE;
c += GEMM_UNROLL_N * ldc * COMPSIZE;
j --;
@@ -298,59 +319,59 @@ int CNAME(BLASLONG m, BLASLONG n, BLASLONG k, FLOAT dummy1,
kk = m + offset;
- i = m % vl;
- if (i) {
- aa = a + (m - i) * k * COMPSIZE;
- cc = c + (m - i) * COMPSIZE;
+ if (m & (GEMM_UNROLL_M - 1)) {
+ for (i = 1; i < GEMM_UNROLL_M; i *= 2){
+ if (m & i) {
+ aa = a + ((m & ~(i - 1)) - i) * k * COMPSIZE;
+ cc = c + ((m & ~(i - 1)) - i) * COMPSIZE;
- if (k - kk > 0) {
- GEMM_KERNEL(i, j, k - kk, dm1,
+ if (k - kk > 0) {
+ GEMM_KERNEL(i, j, k - kk, dm1,
#ifdef COMPLEX
- ZERO,
+ ZERO,
#endif
- aa + i * kk * COMPSIZE,
- b + j * kk * COMPSIZE,
- cc, ldc);
- }
-
- solve(i, j,
- aa + (kk - i) * i * COMPSIZE,
- b + (kk - i) * j * COMPSIZE,
- cc, ldc);
+ aa + i * kk * COMPSIZE,
+ b + j * kk * COMPSIZE,
+ cc, ldc);
+ }
- kk -= i;
+ solve(i, j,
+ aa + (kk - i) * i * COMPSIZE,
+ b + (kk - i) * j * COMPSIZE,
+ cc, ldc);
+ kk -= i;
+ }
+ }
}
- int mod = i;
- i = vl;
- if (i <= m) {
- aa = a + (m - mod - vl) * k * COMPSIZE;
- cc = c + (m - mod - vl) * COMPSIZE;
+ i = (m >> GEMM_UNROLL_M_SHIFT);
+ if (i > 0) {
+ aa = a + ((m & ~(GEMM_UNROLL_M - 1)) - GEMM_UNROLL_M) * k * COMPSIZE;
+ cc = c + ((m & ~(GEMM_UNROLL_M - 1)) - GEMM_UNROLL_M) * COMPSIZE;
do {
if (k - kk > 0) {
- GEMM_KERNEL(vl, j, k - kk, dm1,
+ GEMM_KERNEL(GEMM_UNROLL_M, j, k - kk, dm1,
#ifdef COMPLEX
ZERO,
#endif
- aa + vl * kk * COMPSIZE,
+ aa + GEMM_UNROLL_M * kk * COMPSIZE,
b + j * kk * COMPSIZE,
cc,
ldc);
}
- solve(vl, j,
- aa + (kk - vl) * vl * COMPSIZE,
- b + (kk - vl) * j * COMPSIZE,
+ solve(GEMM_UNROLL_M, j,
+ aa + (kk - GEMM_UNROLL_M) * GEMM_UNROLL_M * COMPSIZE,
+ b + (kk - GEMM_UNROLL_M) * j * COMPSIZE,
cc, ldc);
- aa -= vl * k * COMPSIZE;
- cc -= vl * COMPSIZE;
- kk -= vl;
-
- i += vl;
- } while (i <= m);
+ aa -= GEMM_UNROLL_M * k * COMPSIZE;
+ cc -= GEMM_UNROLL_M * COMPSIZE;
+ kk -= GEMM_UNROLL_M;
+ i --;
+ } while (i > 0);
}
b += j * k * COMPSIZE;
diff --git a/kernel/riscv64/trsm_kernel_LT_rvv_v1.c b/kernel/riscv64/trsm_kernel_LT_rvv_v1.c
index da443cfba..07a2c68a5 100644
--- a/kernel/riscv64/trsm_kernel_LT_rvv_v1.c
+++ b/kernel/riscv64/trsm_kernel_LT_rvv_v1.c
@@ -71,6 +71,30 @@ static FLOAT dm1 = -1.;
#define GEMM_KERNEL GEMM_KERNEL_N
#endif
+#if GEMM_DEFAULT_UNROLL_M == 1
+#define GEMM_UNROLL_M_SHIFT 0
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 2
+#define GEMM_UNROLL_M_SHIFT 1
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 4
+#define GEMM_UNROLL_M_SHIFT 2
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 6
+#define GEMM_UNROLL_M_SHIFT 2
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 8
+#define GEMM_UNROLL_M_SHIFT 3
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 16
+#define GEMM_UNROLL_M_SHIFT 4
+#endif
+
#if GEMM_DEFAULT_UNROLL_N == 1
#define GEMM_UNROLL_N_SHIFT 0
#endif
@@ -91,7 +115,9 @@ static FLOAT dm1 = -1.;
#define GEMM_UNROLL_N_SHIFT 4
#endif
-// Optimizes the implementation in ../arm64/trsm_kernel_LT_sve.c
+// Packed A is tiled by GEMM_UNROLL_M (the GEMM micro-kernel / itcopy contract),
+// NOT by runtime VL: tiling by VSETVL_MAX only matches x280 and silently
+// corrupts on other targets (e.g. zvl128b, unroll 8). Only solve() vectorizes.
#ifndef COMPLEX
@@ -213,10 +239,6 @@ int CNAME(BLASLONG m, BLASLONG n, BLASLONG k, FLOAT dummy1,
BLASLONG kk;
BLASLONG i, j;
- size_t vl = VSETVL_MAX;
-
- //fprintf(stderr, "%s , %s, m = %4ld n = %4ld k = %4ld offset = %4ld\n", __FILE__, __FUNCTION__, m, n, k, offset); // Debug
-
j = (n >> GEMM_UNROLL_N_SHIFT);
while (j > 0) {
@@ -225,47 +247,51 @@ int CNAME(BLASLONG m, BLASLONG n, BLASLONG k, FLOAT dummy1,
aa = a;
cc = c;
- i = vl;
+ i = (m >> GEMM_UNROLL_M_SHIFT);
- while (i <= m) {
+ while (i > 0) {
if (kk > 0) {
- GEMM_KERNEL(vl, GEMM_UNROLL_N, kk, dm1,
+ GEMM_KERNEL(GEMM_UNROLL_M, GEMM_UNROLL_N, kk, dm1,
#ifdef COMPLEX
ZERO,
#endif
aa, b, cc, ldc);
}
- solve(vl, GEMM_UNROLL_N,
- aa + kk * vl * COMPSIZE,
+ solve(GEMM_UNROLL_M, GEMM_UNROLL_N,
+ aa + kk * GEMM_UNROLL_M * COMPSIZE,
b + kk * GEMM_UNROLL_N * COMPSIZE,
cc, ldc);
- aa += vl * k * COMPSIZE;
- cc += vl * COMPSIZE;
- kk += vl;
- i += vl;
+ aa += GEMM_UNROLL_M * k * COMPSIZE;
+ cc += GEMM_UNROLL_M * COMPSIZE;
+ kk += GEMM_UNROLL_M;
+ i --;
}
- i = m % vl;
- if (i) {
- if (kk > 0) {
- GEMM_KERNEL(i, GEMM_UNROLL_N, kk, dm1,
+ if (m & (GEMM_UNROLL_M - 1)) {
+ i = (GEMM_UNROLL_M >> 1);
+ while (i > 0) {
+ if (m & i) {
+ if (kk > 0) {
+ GEMM_KERNEL(i, GEMM_UNROLL_N, kk, dm1,
#ifdef COMPLEX
- ZERO,
+ ZERO,
#endif
- aa, b, cc, ldc);
- }
- solve(i, GEMM_UNROLL_N,
- aa + kk * i * COMPSIZE,
- b + kk * GEMM_UNROLL_N * COMPSIZE,
- cc, ldc);
-
- aa += i * k * COMPSIZE;
- cc += i * COMPSIZE;
- kk += i;
+ aa, b, cc, ldc);
+ }
+ solve(i, GEMM_UNROLL_N,
+ aa + kk * i * COMPSIZE,
+ b + kk * GEMM_UNROLL_N * COMPSIZE,
+ cc, ldc);
+ aa += i * k * COMPSIZE;
+ cc += i * COMPSIZE;
+ kk += i;
+ }
+ i >>= 1;
+ }
}
b += GEMM_UNROLL_N * k * COMPSIZE;
@@ -283,11 +309,11 @@ int CNAME(BLASLONG m, BLASLONG n, BLASLONG k, FLOAT dummy1,
aa = a;
cc = c;
- i = vl;
+ i = (m >> GEMM_UNROLL_M_SHIFT);
- while (i <= m) {
+ while (i > 0) {
if (kk > 0) {
- GEMM_KERNEL(vl, j, kk, dm1,
+ GEMM_KERNEL(GEMM_UNROLL_M, j, kk, dm1,
#ifdef COMPLEX
ZERO,
#endif
@@ -297,37 +323,41 @@ int CNAME(BLASLONG m, BLASLONG n, BLASLONG k, FLOAT dummy1,
ldc);
}
- solve(vl, j,
- aa + kk * vl * COMPSIZE,
+ solve(GEMM_UNROLL_M, j,
+ aa + kk * GEMM_UNROLL_M * COMPSIZE,
b + kk * j * COMPSIZE, cc, ldc);
- aa += vl * k * COMPSIZE;
- cc += vl * COMPSIZE;
- kk += vl;
- i += vl;
+ aa += GEMM_UNROLL_M * k * COMPSIZE;
+ cc += GEMM_UNROLL_M * COMPSIZE;
+ kk += GEMM_UNROLL_M;
+ i --;
}
- i = m % vl;
- if (i) {
- if (kk > 0) {
- GEMM_KERNEL(i, j, kk, dm1,
+ if (m & (GEMM_UNROLL_M - 1)) {
+ i = (GEMM_UNROLL_M >> 1);
+ while (i > 0) {
+ if (m & i) {
+ if (kk > 0) {
+ GEMM_KERNEL(i, j, kk, dm1,
#ifdef COMPLEX
- ZERO,
+ ZERO,
#endif
- aa,
- b,
- cc,
- ldc);
+ aa,
+ b,
+ cc,
+ ldc);
+ }
+
+ solve(i, j,
+ aa + kk * i * COMPSIZE,
+ b + kk * j * COMPSIZE, cc, ldc);
+
+ aa += i * k * COMPSIZE;
+ cc += i * COMPSIZE;
+ kk += i;
+ }
+ i >>= 1;
}
-
- solve(i, j,
- aa + kk * i * COMPSIZE,
- b + kk * j * COMPSIZE, cc, ldc);
-
- aa += i * k * COMPSIZE;
- cc += i * COMPSIZE;
- kk += i;
-
}
b += j * k * COMPSIZE;
diff --git a/kernel/riscv64/trsm_kernel_RN_rvv_v1.c b/kernel/riscv64/trsm_kernel_RN_rvv_v1.c
index 32e481036..c8c61cef6 100644
--- a/kernel/riscv64/trsm_kernel_RN_rvv_v1.c
+++ b/kernel/riscv64/trsm_kernel_RN_rvv_v1.c
@@ -68,6 +68,30 @@ static FLOAT dm1 = -1.;
#define GEMM_KERNEL GEMM_KERNEL_N
#endif
+#if GEMM_DEFAULT_UNROLL_M == 1
+#define GEMM_UNROLL_M_SHIFT 0
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 2
+#define GEMM_UNROLL_M_SHIFT 1
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 4
+#define GEMM_UNROLL_M_SHIFT 2
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 6
+#define GEMM_UNROLL_M_SHIFT 2
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 8
+#define GEMM_UNROLL_M_SHIFT 3
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 16
+#define GEMM_UNROLL_M_SHIFT 4
+#endif
+
#if GEMM_DEFAULT_UNROLL_N == 1
#define GEMM_UNROLL_N_SHIFT 0
#endif
@@ -88,7 +112,9 @@ static FLOAT dm1 = -1.;
#define GEMM_UNROLL_N_SHIFT 4
#endif
-// Optimizes the implementation in ../arm64/trsm_kernel_RN_sve.c
+// Packed A is tiled by GEMM_UNROLL_M (the GEMM micro-kernel / itcopy contract),
+// NOT by runtime VL: tiling by VSETVL_MAX only matches x280 and silently
+// corrupts on other targets (e.g. zvl128b, unroll 8). Only solve() vectorizes.
#ifndef COMPLEX
@@ -209,11 +235,6 @@ int CNAME(BLASLONG m, BLASLONG n, BLASLONG k, FLOAT dummy1,
BLASLONG kk;
BLASLONG i, j;
- size_t vl = VSETVL_MAX;
-
- //fprintf(stderr, "%s , %s, m = %4ld n = %4ld k = %4ld offset = %4ld\n", __FILE__, __FUNCTION__, m, n, k, offset); // Debug
-
-
j = (n >> GEMM_UNROLL_N_SHIFT);
kk = -offset;
@@ -222,47 +243,50 @@ int CNAME(BLASLONG m, BLASLONG n, BLASLONG k, FLOAT dummy1,
aa = a;
cc = c;
- i = vl;
+ i = (m >> GEMM_UNROLL_M_SHIFT);
- if (i <= m) {
+ if (i > 0) {
do {
- if (kk > 0) {
- GEMM_KERNEL(vl, GEMM_UNROLL_N, kk, dm1,
+ if (kk > 0) {
+ GEMM_KERNEL(GEMM_UNROLL_M, GEMM_UNROLL_N, kk, dm1,
#ifdef COMPLEX
- ZERO,
+ ZERO,
#endif
- aa, b, cc, ldc);
- }
-
- solve(vl, GEMM_UNROLL_N,
- aa + kk * vl * COMPSIZE,
- b + kk * GEMM_UNROLL_N * COMPSIZE,
- cc, ldc);
-
- aa += vl * k * COMPSIZE;
- cc += vl * COMPSIZE;
- i += vl;
- } while (i <= m);
- }
+ aa, b, cc, ldc);
+ }
+ solve(GEMM_UNROLL_M, GEMM_UNROLL_N,
+ aa + kk * GEMM_UNROLL_M * COMPSIZE,
+ b + kk * GEMM_UNROLL_N * COMPSIZE,
+ cc, ldc);
+
+ aa += GEMM_UNROLL_M * k * COMPSIZE;
+ cc += GEMM_UNROLL_M * COMPSIZE;
+ i --;
+ } while (i > 0);
+ }
- i = m % vl;
- if (i) {
- if (kk > 0) {
- GEMM_KERNEL(i, GEMM_UNROLL_N, kk, dm1,
+ if (m & (GEMM_UNROLL_M - 1)) {
+ i = (GEMM_UNROLL_M >> 1);
+ while (i > 0) {
+ if (m & i) {
+ if (kk > 0) {
+ GEMM_KERNEL(i, GEMM_UNROLL_N, kk, dm1,
#ifdef COMPLEX
- ZERO,
+ ZERO,
#endif
- aa, b, cc, ldc);
+ aa, b, cc, ldc);
+ }
+ solve(i, GEMM_UNROLL_N,
+ aa + kk * i * COMPSIZE,
+ b + kk * GEMM_UNROLL_N * COMPSIZE,
+ cc, ldc);
+
+ aa += i * k * COMPSIZE;
+ cc += i * COMPSIZE;
+ }
+ i >>= 1;
}
- solve(i, GEMM_UNROLL_N,
- aa + kk * i * COMPSIZE,
- b + kk * GEMM_UNROLL_N * COMPSIZE,
- cc, ldc);
-
- aa += i * k * COMPSIZE;
- cc += i * COMPSIZE;
-
}
kk += GEMM_UNROLL_N;
@@ -277,57 +301,61 @@ int CNAME(BLASLONG m, BLASLONG n, BLASLONG k, FLOAT dummy1,
while (j > 0) {
if (n & j) {
- aa = a;
- cc = c;
+ aa = a;
+ cc = c;
- i = vl;
+ i = (m >> GEMM_UNROLL_M_SHIFT);
- while (i <= m) {
- if (kk > 0) {
- GEMM_KERNEL(vl, j, kk, dm1,
+ while (i > 0) {
+ if (kk > 0) {
+ GEMM_KERNEL(GEMM_UNROLL_M, j, kk, dm1,
#ifdef COMPLEX
- ZERO,
+ ZERO,
#endif
- aa,
- b,
- cc,
- ldc);
- }
-
- solve(vl, j,
- aa + kk * vl * COMPSIZE,
- b + kk * j * COMPSIZE, cc, ldc);
-
- aa += vl * k * COMPSIZE;
- cc += vl * COMPSIZE;
- i += vl;
- }
-
- i = m % vl;
- if (i) {
- if (kk > 0) {
- GEMM_KERNEL(i, j, kk, dm1,
+ aa,
+ b,
+ cc,
+ ldc);
+ }
+
+ solve(GEMM_UNROLL_M, j,
+ aa + kk * GEMM_UNROLL_M * COMPSIZE,
+ b + kk * j * COMPSIZE, cc, ldc);
+
+ aa += GEMM_UNROLL_M * k * COMPSIZE;
+ cc += GEMM_UNROLL_M * COMPSIZE;
+ i --;
+ }
+
+ if (m & (GEMM_UNROLL_M - 1)) {
+ i = (GEMM_UNROLL_M >> 1);
+ while (i > 0) {
+ if (m & i) {
+ if (kk > 0) {
+ GEMM_KERNEL(i, j, kk, dm1,
#ifdef COMPLEX
- ZERO,
+ ZERO,
#endif
- aa,
- b,
- cc,
- ldc);
- }
-
- solve(i, j,
- aa + kk * i * COMPSIZE,
- b + kk * j * COMPSIZE, cc, ldc);
-
- aa += i * k * COMPSIZE;
- cc += i * COMPSIZE;
-
- }
+ aa,
+ b,
+ cc,
+ ldc);
+ }
+
+ solve(i, j,
+ aa + kk * i * COMPSIZE,
+ b + kk * j * COMPSIZE, cc, ldc);
+
+ aa += i * k * COMPSIZE;
+ cc += i * COMPSIZE;
+ }
+ i >>= 1;
+ }
+ }
- b += j * k * COMPSIZE;
- c += j * ldc * COMPSIZE;
- kk += j;
+ b += j * k * COMPSIZE;
+ c += j * ldc * COMPSIZE;
+ kk += j;
}
j >>= 1;
}
diff --git a/kernel/riscv64/trsm_kernel_RT_rvv_v1.c b/kernel/riscv64/trsm_kernel_RT_rvv_v1.c
index 81cc41818..dd0eb6a74 100644
--- a/kernel/riscv64/trsm_kernel_RT_rvv_v1.c
+++ b/kernel/riscv64/trsm_kernel_RT_rvv_v1.c
@@ -67,6 +67,30 @@ static FLOAT dm1 = -1.;
#define GEMM_KERNEL GEMM_KERNEL_N
#endif
+#if GEMM_DEFAULT_UNROLL_M == 1
+#define GEMM_UNROLL_M_SHIFT 0
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 2
+#define GEMM_UNROLL_M_SHIFT 1
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 4
+#define GEMM_UNROLL_M_SHIFT 2
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 6
+#define GEMM_UNROLL_M_SHIFT 2
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 8
+#define GEMM_UNROLL_M_SHIFT 3
+#endif
+
+#if GEMM_DEFAULT_UNROLL_M == 16
+#define GEMM_UNROLL_M_SHIFT 4
+#endif
+
#if GEMM_DEFAULT_UNROLL_N == 1
#define GEMM_UNROLL_N_SHIFT 0
#endif
@@ -87,7 +111,9 @@ static FLOAT dm1 = -1.;
#define GEMM_UNROLL_N_SHIFT 4
#endif
-// Optimizes the implementation in ../arm64/trsm_kernel_RT_sve.c
+// Packed A is tiled by GEMM_UNROLL_M (the GEMM micro-kernel / itcopy contract),
+// NOT by runtime VL: tiling by VSETVL_MAX only matches x280 and silently
+// corrupts on other targets (e.g. zvl128b, unroll 8). Only solve() vectorizes.
#ifndef COMPLEX
@@ -215,10 +241,6 @@ int CNAME(BLASLONG m, BLASLONG n, BLASLONG k, FLOAT dummy1,
FLOAT *aa, *cc;
BLASLONG kk;
- size_t vl = VSETVL_MAX;
-
- //fprintf(stderr, "%s , %s, m = %4ld n = %4ld k = %4ld offset = %4ld\n", __FILE__, __FUNCTION__, m, n, k, offset); // Debug
-
kk = n - offset;
c += n * ldc * COMPSIZE;
b += n * k * COMPSIZE;
@@ -234,52 +256,58 @@ int CNAME(BLASLONG m, BLASLONG n, BLASLONG k, FLOAT dummy1,
c -= j * ldc* COMPSIZE;
cc = c;
- i = vl;
- if (i <= m) {
+ i = (m >> GEMM_UNROLL_M_SHIFT);
+ if (i > 0) {
do {
if (k - kk > 0) {
- GEMM_KERNEL(vl, j, k - kk, dm1,
+ GEMM_KERNEL(GEMM_UNROLL_M, j, k - kk, dm1,
#ifdef COMPLEX
ZERO,
#endif
- aa + vl * kk * COMPSIZE,
+ aa + GEMM_UNROLL_M * kk * COMPSIZE,
b + j * kk * COMPSIZE,
cc,
ldc);
}
- solve(vl, j,
- aa + (kk - j) * vl * COMPSIZE,
+ solve(GEMM_UNROLL_M, j,
+ aa + (kk - j) * GEMM_UNROLL_M * COMPSIZE,
b + (kk - j) * j * COMPSIZE,
cc, ldc);
- aa += vl * k * COMPSIZE;
- cc += vl * COMPSIZE;
- i += vl;
- } while (i <= m);
+ aa += GEMM_UNROLL_M * k * COMPSIZE;
+ cc += GEMM_UNROLL_M * COMPSIZE;
+ i --;
+ } while (i > 0);
}
- i = m % vl;
- if (i) {
- if (k - kk > 0) {
- GEMM_KERNEL(i, j, k - kk, dm1,
+ if (m & (GEMM_UNROLL_M - 1)) {
+ i = (GEMM_UNROLL_M >> 1);
+ do {
+ if (m & i) {
+
+ if (k - kk > 0) {
+ GEMM_KERNEL(i, j, k - kk, dm1,
#ifdef COMPLEX
- ZERO,
+ ZERO,
#endif
- aa + i * kk * COMPSIZE,
- b + j * kk * COMPSIZE,
- cc, ldc);
- }
+ aa + i * kk * COMPSIZE,
+ b + j * kk * COMPSIZE,
+ cc, ldc);
+ }
- solve(i, j,
- aa + (kk - j) * i * COMPSIZE,
- b + (kk - j) * j * COMPSIZE,
- cc, ldc);
+ solve(i, j,
+ aa + (kk - j) * i * COMPSIZE,
+ b + (kk - j) * j * COMPSIZE,
+ cc, ldc);
- aa += i * k * COMPSIZE;
- cc += i * COMPSIZE;
+ aa += i * k * COMPSIZE;
+ cc += i * COMPSIZE;
+ }
+ i >>= 1;
+ } while (i > 0);
}
kk -= j;
}
@@ -297,52 +325,56 @@ int CNAME(BLASLONG m, BLASLONG n, BLASLONG k, FLOAT dummy1,
c -= GEMM_UNROLL_N * ldc * COMPSIZE;
cc = c;
- i = vl;
- if (i <= m) {
- do {
- if (k - kk > 0) {
- GEMM_KERNEL(vl, GEMM_UNROLL_N, k - kk, dm1,
+ i = (m >> GEMM_UNROLL_M_SHIFT);
+ if (i > 0) {
+ do {
+ if (k - kk > 0) {
+ GEMM_KERNEL(GEMM_UNROLL_M, GEMM_UNROLL_N, k - kk, dm1,
#ifdef COMPLEX
- ZERO,
+ ZERO,
#endif
- aa + vl * kk * COMPSIZE,
- b + GEMM_UNROLL_N * kk * COMPSIZE,
- cc,
- ldc);
- }
-
- solve(vl, GEMM_UNROLL_N,
- aa + (kk - GEMM_UNROLL_N) * vl * COMPSIZE,
- b + (kk - GEMM_UNROLL_N) * GEMM_UNROLL_N * COMPSIZE,
- cc, ldc);
-
- aa += vl * k * COMPSIZE;
- cc += vl * COMPSIZE;
- i += vl;
- } while (i <= m);
+ aa + GEMM_UNROLL_M * kk * COMPSIZE,
+ b + GEMM_UNROLL_N * kk * COMPSIZE,
+ cc,
+ ldc);
+ }
+
+ solve(GEMM_UNROLL_M, GEMM_UNROLL_N,
+ aa + (kk - GEMM_UNROLL_N) * GEMM_UNROLL_M * COMPSIZE,
+ b + (kk - GEMM_UNROLL_N) * GEMM_UNROLL_N * COMPSIZE,
+ cc, ldc);
+
+ aa += GEMM_UNROLL_M * k * COMPSIZE;
+ cc += GEMM_UNROLL_M * COMPSIZE;
+ i --;
+ } while (i > 0);
}
- i = m % vl;
- if (i) {
- if (k - kk > 0) {
- GEMM_KERNEL(i, GEMM_UNROLL_N, k - kk, dm1,
+ if (m & (GEMM_UNROLL_M - 1)) {
+ i = (GEMM_UNROLL_M >> 1);
+ do {
+ if (m & i) {
+ if (k - kk > 0) {
+ GEMM_KERNEL(i, GEMM_UNROLL_N, k - kk, dm1,
#ifdef COMPLEX
- ZERO,
+ ZERO,
#endif
- aa + i * kk * COMPSIZE,
- b + GEMM_UNROLL_N * kk * COMPSIZE,
- cc,
- ldc);
- }
-
- solve(i, GEMM_UNROLL_N,
- aa + (kk - GEMM_UNROLL_N) * i * COMPSIZE,
- b + (kk - GEMM_UNROLL_N) * GEMM_UNROLL_N * COMPSIZE,
- cc, ldc);
+ aa + i * kk * COMPSIZE,
+ b + GEMM_UNROLL_N * kk * COMPSIZE,
+ cc,
+ ldc);
+ }
- aa += i * k * COMPSIZE;
- cc += i * COMPSIZE;
+ solve(i, GEMM_UNROLL_N,
+ aa + (kk - GEMM_UNROLL_N) * i * COMPSIZE,
+ b + (kk - GEMM_UNROLL_N) * GEMM_UNROLL_N * COMPSIZE,
+ cc, ldc);
+ aa += i * k * COMPSIZE;
+ cc += i * COMPSIZE;
+ }
+ i >>= 1;
+ } while (i > 0);
}
kk -= GEMM_UNROLL_N; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
trsm_kernel_{LN,LT,RN,RT}_rvv_v1.c) forRISCV64_ZVL128Btargetx280target, but were not enabled for ZVL128B configurationNote on ZVL256B
ZVL256B is intentionally excluded from this change. Testing revealed that the RVV TRSM kernel produces incorrect results with VLEN=256 (e.g., 3x3 lower triangular solve returns wrong values). This needs to be investigated separately.
Test plan